In this work we show obstacles when constructing identity-based encryption (IBE) from isogenies. We first give a modular description for IBEs, what we call a canonical IBE, that consists of two components: an identity key derivation scheme and a public-key encryption scheme. This allows us to investigate the identity key derivation scheme (where the obstacles are rooted in) in isolation. We present several approaches, showing that they can either not be realized—extracting the secret keys would require to solve the underlying hardness assumption—or result in IBE schemes that are insecure—users can use their secret keys to compute secret keys of other users. Finally, we identify properties for isogeny-based trapdoor functions that one would need in order to overcome the identified obstacles.