Testing 5d-6d dualities with fractional D-branes

6d SCFTs compactified on a circle can often be studied from nonperturbative 5d super-Yang-Mills theories, using instanton solitons. However, the 5d Yang-Mills theories with 6d UV fixed points frequently have too many hypermultiplet matters, which makes it difficult to use the ADHM techniques for instantons. With the examples of 6d $\mathcal{N}=(1,0)$ SCFTs with $Sp(N)$ gauge symmetry and $2N+8$ fundamental hypermultiplets, we show that one can still make rigorous studies of these 5d-6d relations in the `fractional D-brane sectors'. We test the recently proposed 5d duals given by $Sp(N+1)$ gauge theories, and compare their instanton partition functions with the elliptic genera of 6d self-dual strings.