CTF parameters from cryosparc

[GenevieveBuckley]

I notice there is a function cryosparc_2_cs_ctf_parameters that exists in the repository, but is never actually executed. I'd expected to see the function be called somewhere here, but it isn't. The function was added in this commit "Converting everything but Zernike coefs, and I haven't been able to find any other context.

pyem/pyem/metadata.py

Lines 291 to 312 in 4af3869

	def cryosparc_2_cs_ctf_parameters(cs, df=None):
	log = logging.getLogger('root')
	if df is None:
	df = pd.DataFrame()
	if 'ctf/tilt_A' in cs.dtype.names:
	log.debug("Recovering beam tilt")
	df[star.Relion.BEAMTILTX] = cs['ctf/tilt_A'][:, 0]
	df[star.Relion.BEAMTILTY] = cs['ctf/tilt_A'][:, 1]
	if 'ctf/shift_A' in cs.dtype.names:
	pass
	if 'ctf/trefoil_A' in cs.dtype.names:
	pass
	# df[star.Relion.ODDZERNIKE] = cs['ctf/trefoil_A']
	if 'ctf/tetrafoil_A' in cs.dtype.names:
	pass
	# df[star.Relion.EVENZERNIKE] = cs['ctf/tetra_A']
	if 'ctf/anisomag' in cs.dtype.names:
	df[star.Relion.MAGMAT00] = cs['ctf/anisomag'][:, 0]
	df[star.Relion.MAGMAT01] = cs['ctf/anisomag'][:, 1]
	df[star.Relion.MAGMAT10] = cs['ctf/anisomag'][:, 2]
	df[star.Relion.MAGMAT11] = cs['ctf/anisomag'][:, 3]
	return df

Can you tell me more about it?

Context: A colleague of mine asked if it was possible to transfer the CTF parameter metadata when converting files from CryoSPARC back to Relion. That's how I found this function. It looks like it does some of what we want, but we are currently struggling to figure out exactly how to make the conversion to the Zernike parameters Relion needs.

[asarnow]

I'm sorry @GenevieveBuckley, I forgot to reply earlier - thank you for your interest!

As you noticed, I didn't finish implementing this conversion function. The simplest part is the anisotropic magnification, as the matrix elements are unitless. The matrix conversion is untested, but I believe cryoSPARC and Relion use the same axis convention for particles. If it doesn't work than transposing the matrix is probably needed (or swapping the columns, or exchanging 10 and 11).

For beam tilts, there is a unit conversion needed - Relion uses mrad while csparc gives it in Å. That means we need a characteristic length, which turns out to be the Cs. I just committed that part in 0b4d7d2.

Unfortunately, I don't know how to convert to Zernike coefficients. It would be awesome if we could figure it out though!

[GenevieveBuckley]

I'm looking at "Principles of Electron Optics" by P. W. Hawkes now, in the hopes it explains some of the equations I'll need. There's only one mention of "Zernike" in the index though, so this might not be the best place to look. You don't happen to know a better resource for the maths, do you?

Right now I'm at the complete beginning - first I'd need to find which equation describes things for Relion, and which equation CryoSPARC uses, then figure out which parameters each of the numbers in the metadata match to. And then after that we can look at how you might convert between the two.

[asarnow]

Awesome - one of my favorite subjects. Everything we need is in two papers describing the original beam tilt implementation [1] and the Zernike polynomial expansion [2] in Relion. With the Relion docs [3] and Wikipedia article for Zernike polynomials [4] as well I think I have it almost figured out.

There are 6 odd Zernike coefficients that derive from shift_A, tilt_A, and trefoil_A (all 2-vectors). We will need a unit conversion of 2π for the shift, and of 2π * Cs * λ^2 for the other two (e.g. [1] Eqn 5). For Relion these 6 values must become a comma separated list in the proper order under rlnOddZernike [3]. Based on [4], shift_A => Z(-1,1), Z(1,1), tilt_A => Z(-1,3), Z(1,3), and trefoil_A => Z(-3,3), Z(3,3).

Then there are 9 even Zernike coefficients that derive from the total amplitude contrast/phase shift, the defocus matrix, the Cs, and tetrafoil astigmatism (tetra_A, 4-vector). Again we need a comma-separated list of the coefficients, to put in rlnEvenZernike. The phase shift is AC - arccos(phi) and corresponds to Z(0,0). There is a unit factor of πλ for the unique parts of the defocus matrix elements (dz, a1 and a2 per the Relion terminology), corresponding to Z(0, 2) for avg. defocus, and Z(-2,2), Z(2,2) for astigmatism. Finally the unit conversion for the Cs and tetrafoil parts will be -π/2 * λ^3, these correspond to Z(0,4) and Z(-4,4), Z(-2,4), Z(2,4), and Z(4,4) respectively.

The ctf_parameters() method could make the conversions and set columns in the particles DataFrame for each coefficient (using new constants like star.Relion.Z_0_0 = "rlnZernike_0_0", etc). Then these fields can be joined with a comma to make new string columns for star.Relion.ODDZERNIKE and star.Relion.EVENZERNIKE.

Does that make sense to you? If you're willing to help implement this (which I would very, very much appreciate) it might be easier to make a plan together in another format. (E.g. Discord).

[1] https://elifesciences.org/articles/42166
[2] https://journals.iucr.org/m/issues/2020/02/00/fq5009/
[3] https://relion.readthedocs.io/en/release-3.1/Reference/Conventions.html
[4] https://en.wikipedia.org/wiki/Zernike_polynomials#Zernike_polynomials

[GenevieveBuckley]

Does that make sense to you?

It's starting to make a lot more sense, thank you! (I looked at [2] and [3] earlier, but had a hard time without the extra context. Re-reading now, I'm starting to understand the pieces more.)

If you're willing to help implement this (which I would very, very much appreciate) it might be easier to make a plan together in another format. (E.g. Discord).

Yes, I do want to do that. I'll send an email to the address you have listed on your GitHub profile, and we can set up a time to talk.

[GenevieveBuckley]

Email sent!

[GenevieveBuckley]

Some rough work is available here: #87

[GenevieveBuckley]

#87 is merged, so I guess I should close this issue Edit: nope, it was merged into a different branch for testing, so we'll keep this issue open.

[olibclarke]

Came across this while looking to see if it was possible to translate CTF params from cryosparc to relion - it would definitely be a major enhancement if this was folded into csparc2star.py, even if it only converted beam tilts and ignored other parameters

[GenevieveBuckley]

Came across this while looking to see if it was possible to translate CTF params from cryosparc to relion - it would definitely be a major enhancement if this was folded into csparc2star.py, even if it only converted beam tilts and ignored other parameters

This work was paused because we're not confident about the accuracy (see here for details). If you're able to help track down the source of the mismatch, I'm sure that would be very welcome.