Prime twists of elliptic curves

For certain elliptic curves $E/\mathbb{Q}$ with $E(\mathbb{Q})[2]=\mathbb{Z}/2 \mathbb{Z}$, we prove a criterion for prime twists of $E$ to have analytic rank 0 or 1, based on a mod 4 congruence of 2-adic logarithms of Heegner points. As an application, we prove new cases of Silverman's conjecture that there exists a positive proposition of prime twists of $E$ of rank zero (resp. positive rank).