Developments in sensor technology, data acquisition and data transfer allow us to monitor structural behavior in a detailed way, in space as well as over time. At the same time, simulation tools enable us to build detailed computational models which closely mimic the behavior of real structures. The level of detail in simulation and sensing technologies naturally calls for a fusion of experimental data and numerical simulations in a digital twin of a structure. Even when many of the data processing and updating methods required to feed a digital twin are considered mature, challenges still arise when applying these to actual structures as will be highlighted in two case studies. The first case study involves a steel railway bridge of the bowstring type which was monitored for a period of six years to validate virtual sensing methods for fatigue assessment. The consideration of the behavior of the bearings turned out to be crucial for the virtual sensing to be successful. The second case study involves a post-tensioned concrete girder highway bridge which dates from the fifties and shows signs of structural deterioration. This bridge was monitored over a period of two years by FBG sensors measuring quasi-static as well as (periodically) dynamic strains while it was also subjected to two conventional load tests, one right after construction, another more recent. This work is currently still under development and some of the challenges encountered so far in integrating and fusing the information for updating a model will be discussed.