We examine the sensitiveness of the free-energy landscape of a directed polymer in random media with respect to various kinds of infinitesimally weak perturbation including the intriguing case of temperature chaos. To this end, we combine the replica Bethe Ansatz approach outlined by Sales and Yoshino (e-print cond-mat/0112384), the mapping to a modified Sinai model, and numerically exact calculations by the transfer-matrix method. Our results imply that for all the perturbations under study there is a slow crossover from a weakly perturbed regime, where rare events take place, to a strongly perturbed regime at larger length scales beyond the so-called overlap length, where typical events take place leading to chaos, i.e., a complete reshuffling of the free-energy landscape. Within the replica space, the evidence for chaos is found in the factorization of the replicated partition function induced by infinitesimal perturbations. This is the reflex of explicit replica-symmetry breaking.