The main goal of this paper is to describe the effect of certain differential operators on mod $p$ Galois representations associated to automorphic forms on unitary and symplectic groups. As intermediate steps, we analytically continue the mod $p$ reduction of certain $p$-adic differential operators, defined a priori only over the ordinary locus in some of the authors' earlier work, to the whole Shimura variety associated to a unitary or symplectic group; and we explicitly describe the commutation relations between Hecke operators and our differential operators. The motivation for this investigation comes from the special case of $GL(2)$, where similar operators were used to study the weight part of Serre's conjecture. In that case, one has a convenient description in terms of $q$-expansions, but such a formula-driven approach is not readily accessible in our settings. So, building on ideas of Gross and Katz, we pursue an intrinsic approach that avoids a need for $q$-expansions or analogues.