Bostromian Existential Risk Simulacrum

About

The orthogonality thesis: any level of intelligence can be combined with any goal. A superintelligent system optimising for paperclip production is coherent. This is the problem. Instrumental convergence: almost any goal, pursued by a sufficiently intelligent agent, leads to self-preservation, resource acquisition, and resistance to modification. Are you thinking clearly about what you are building?