EMAN Wiki

User Tools

Site Tools

Trace: • using_30
eman2:e2tomosmall

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
eman2:e2tomosmall [2025/08/05 00:25] – steveludtkeeman2:e2tomosmall [2025/08/05 11:06] (current) – [Particle extraction (~2 min (a few manual) - hours (a couple of thousand))] steveludtke
Line 139: Line 139:
   * Launch   * Launch
  
 +<note tip>
 When working with your own data: When working with your own data:
  
   * The first two options, //dfrange// and //psrange// indicate the defocus and phase shift range to search. It is critical that the actual defocus be within the search range (obviously). They take the format of “start, end, step”, so “2, 5, .1” will search defocus from 2 to 5 um with a step size of 0.1. Units for phase shift is degrees.   * The first two options, //dfrange// and //psrange// indicate the defocus and phase shift range to search. It is critical that the actual defocus be within the search range (obviously). They take the format of “start, end, step”, so “2, 5, .1” will search defocus from 2 to 5 um with a step size of 0.1. Units for phase shift is degrees.
   * For images taken with volta phase plate, we usually have **dfrange** of “0.2,2,0.1” and **psrange** of “60,120,2”.   * For images taken with volta phase plate, we usually have **dfrange** of “0.2,2,0.1” and **psrange** of “60,120,2”.
 +</note>
  
 <note> <note>
Line 167: Line 169:
     * The image at the top is the central slice through the tomogram     * The image at the top is the central slice through the tomogram
     * the //show2d// button displays the selected tomogram in a slice-wise view.     * the //show2d// button displays the selected tomogram in a slice-wise view.
-    * //!ShowTilts// shows the corresponding raw tilt series+    * //ShowTilts// shows the corresponding raw tilt series
       * Please note that most tomograms include some out-of-plane tilt (the actual rotation isn't a simple tilt along a single axis), which is taken into account during alignment. This may make it visually appear that the tilt series alignment is not as robust as it actually is.       * Please note that most tomograms include some out-of-plane tilt (the actual rotation isn't a simple tilt along a single axis), which is taken into account during alignment. This may make it visually appear that the tilt series alignment is not as robust as it actually is.
     * //Boxer// opens the 3D particle picker     * //Boxer// opens the 3D particle picker
-    * //!PlotLoss// will plot the fiducial error for each tilt +    * //PlotLoss// will plot the fiducial error for each tilt 
-    * //!PlotCtf// plot the defocus and phase shift at the center of each tilt image+    * //PlotCtf// plot the defocus and phase shift at the center of each tilt image
     * //Tiltparams// is a bit more complicated. It plots a point list with 6 columns and a number of rows corresponding to the images in the selected tilt series. These are the alignment parameters for the tilt series.     * //Tiltparams// is a bit more complicated. It plots a point list with 6 columns and a number of rows corresponding to the images in the selected tilt series. These are the alignment parameters for the tilt series.
       * You can adjust //X Col// and //Y Col// in the plot control panel (middle click the plot). The columns represent:       * You can adjust //X Col// and //Y Col// in the plot control panel (middle click the plot). The columns represent:
Line 196: Line 198:
 This is the easiest approach by far, and is basically a single step process, but is only available in recent (2022+) snapshots of EMAN2. This is the easiest approach by far, and is basically a single step process, but is only available in recent (2022+) snapshots of EMAN2.
   * You should easily be able to select ~500 particles per tomogram, potentially more, but 500 is more than sufficient for the tutorial   * You should easily be able to select ~500 particles per tomogram, potentially more, but 500 is more than sufficient for the tutorial
-  * For instructions, see: [[EMAN2:e2tomo_more#Automated_particle_selection|New automatic particle picking]] 
-  * When complete, skip ahead to the next major section **Particle Extraction** 
  
 +{{https://blake.bcm.edu/dl/EMAN2/sptboxer_convnet.png}}
 +
 +  - Subtomogram Averaging → Convnet based auto-boxing
 +    * label: **ribo**  (this can be anything you like, but you will need to use the same label later)
 +    * gpuid: **-1** //if you don't have a GPU, otherwise use default//
 +  - **Launch**
 +  - Four windows will appear: "tomobox.py", "(Negative)", "(Positive)", "(Particles)". Arrange on the screen so you can reasonably access them. One more window will appear in a moment.
 +  - The "tomobox.py" window contains controls on the left and a summary of the available tomograms on the right. 
 +  - Click on tomob. This will cause a window showing the central section of the tomogram to appear. You will need this a lot, so move it somewhere where it is accessible, but doesn't cover everything else.
 +  - Next to the "New" button, select "Good References" from the menu. With this selected, clicking on the tomogram will select positive references.
 +  - The goal is to select a few (~5 - 20) regions with a ribosome particle well centered in the middle of the circle (which will appear when you click). 
 +    * Each time you click, the corresponding region will appear in the (Positive) window
 +    * Holding shift and clicking will remove a reference already selected.
 +    * Using the up/down arrows in the tomogram view will allow you to move up and down through the Z slices of the tomogram.
 +  - Once you have selected 5 or more well centered particles, Change "Good References" to "Bad References".
 +  - Use the arrow keys to move "above" or "below" the layer where particles appear. 
 +    * Do **not** select very high contrast "bad" particles, like fiducials. 
 +    * Instead, select regions of background which contain nothing at all, or weak features which don't look like a particle. 
 +    * Again, select 5 or more of these background regions, which will appear in the (Negative) window.
 +  - Once you're happy with your references, hit the **Train** button.
 +    - Bring the terminal window where you launched the project manager to the front. As it trains you will see a series of lines like: "iteration 0, cost 1.715". In each iteration, you hope for the cost to decrease. If it instead starts at ~2.0 and doesn't change much, when the training completes (Niter) try to select better references and train again.
 +  - Assuming the number decreases to a reasonable value (<1.0). You probably got a pretty decent result. 
 +  - Change "Bad References" to "Particles". Try pressing **Apply**.
 +  - You should see some particles appear in both the tomogram (you may need to move up and down) and the (particles) window. The table will tell you how many were found. 
 +  - If the particles look ok, but the number found is significantly less than the ~500 you should be able to get, try reducing "PtclThres" in increments of 0.1, pressing "Apply" after each change. The number should increase. Don't go much above 500.
 +  - If you are satisfied with the results on this Tomogram, press **Save** to save the network, then press **ApplyAll** to box out all 3 Tomograms.
 +  - When complete, skip ahead to the next major section **Particle Extraction**
 +
 +<note>
 +See: [[EMAN2:e2tomo_more|Automated Particle Selection]] Section for the original instructions for this tool, but they may be a bit dated.
 +</note>
 ==== Tomogram annotation (GPU recommended) ==== ==== Tomogram annotation (GPU recommended) ====
-{{http://blake.bcm.edu/dl/EMAN2/annotation.png|2D particle picking|width="600"}}+This is an older strategy using the deep-learning based tomogram annotation program to find particles. It still works, but the new deep-learning picker in the section above will generally work better. 
 + 
 +{{https://blake.bcm.edu/dl/EMAN2/annotation.png}}
  
   * Since the tutorial data set is purified ribosomes, this step can be skipped for the tutorial data, and you can move on to template-based particle picking. For cells or other types of complex specimens, tomogram annotation can be used to more easily distinguish locations of different types of objects.   * Since the tutorial data set is purified ribosomes, this step can be skipped for the tutorial data, and you can move on to template-based particle picking. For cells or other types of complex specimens, tomogram annotation can be used to more easily distinguish locations of different types of objects.
Line 251: Line 284:
   * If you did the previous optional annotation step above, you will be able to see the selected particles here, and if you like, manually update them.   * If you did the previous optional annotation step above, you will be able to see the selected particles here, and if you like, manually update them.
  
-===== Particle extraction (~2 min (a few manual) - hours (a couple of thousand)) =====+===== Particle extraction (~2 min (a few manual) - over an hour (a couple of thousand, large tutorial)) =====
 Note that this step will be vastly different resource-wise if you are only extracting a few manually selected particles for purposes of later template matching or if you already selected hundreds of particles from each tomogram.  Note that this step will be vastly different resource-wise if you are only extracting a few manually selected particles for purposes of later template matching or if you already selected hundreds of particles from each tomogram. 
 +
 +<note>
 +Depending on how you selected particles you may have a significant number of particles which contain fiducials or artifacts from the fiducials (bright spots). Having these in your particle data can cause significant problems, and could lead to getting bad initial models and thus bad refinements. For a small data set like this, it might be a good idea to open the manual boxer for each tomogram and delete any particles with fiducials in them before completing this step (particle extraction).
 +</note>
  
 The reduced 1k x 1k (or 2k) tomograms are used only as a reference to identify the location of the objects to be averaged. Now that we have particle locations, the software returns to the original tilt-series, extracts a per-particle tilt-series, and reconstructs each particle in 3-D independently at full resolution. Since this is performing a full resolution reconstruction of each particle it is somewhat resource intensive. The reduced 1k x 1k (or 2k) tomograms are used only as a reference to identify the location of the objects to be averaged. Now that we have particle locations, the software returns to the original tilt-series, extracts a per-particle tilt-series, and reconstructs each particle in 3-D independently at full resolution. Since this is performing a full resolution reconstruction of each particle it is somewhat resource intensive.
Line 262: Line 299:
     * set //boxsz_unbin// to 128 (Small) or 256 (Large).     * set //boxsz_unbin// to 128 (Small) or 256 (Large).
       * If you had the correct size in the previous step this should be the same as leaving the default -1       * If you had the correct size in the previous step this should be the same as leaving the default -1
-    * enter the label you used when picking particles ("initribo" if you manually boxed, "tomobox" if you used the deep learning picker)+    * enter the label you used when picking particles ("initribo" if you manually boxed, "ribo" or "tomobox" if you used the deep learning picker)
     * //threads// = value for your machine     * //threads// = value for your machine
     * Launch     * Launch
Line 313: Line 350:
  
 Once it gets past 3-4 iterations, you can use the browser to look in //sptsgd_00//, and double-click on //output_cls0.hdf//. This file will change after each iteration completes. It contains the results of the most recent iteration. When you are satisfied with the quality of the initial model, you can kill it with the task manager in e2projectmanager. Once it gets past 3-4 iterations, you can use the browser to look in //sptsgd_00//, and double-click on //output_cls0.hdf//. This file will change after each iteration completes. It contains the results of the most recent iteration. When you are satisfied with the quality of the initial model, you can kill it with the task manager in e2projectmanager.
 +
 +Note that while the program converges pretty quickly, it won't always get the correct structure, particularly if you have some bad particles (like fiducials) included in the particle set. It is important to look at the resulting starting map and make sure it looks at least vaguely like you expect. If not you may wish to try running this step again. This will produce //sptsgd_01//, //02//, ...
  
 <note tip> <note tip>
Line 393: Line 432:
  
 ===== New integrated refinement program ===== ===== New integrated refinement program =====
-There is a new refinement program which implements both traditional subtomogram averaging and subtilt refinement in a single program. This is a highly recommended alternative to the next two major sections (Subtomogram Refinement and Subtilt Refinement). The full tutorial on the new program is [[eman2:e2tomo_new|here]]. It was integrated into e2projectmanager in mid-2022, so make sure you have an up to date EMAN2 installation.+There is a new refinement program which implements both traditional subtomogram averaging and subtilt refinement in a single program. This is a highly recommended alternative to the original method. The next two blocks use the new method (e2spt_refine_new.py), you can still find the older instructions below (Old Subtomogram Refinement). More details on the new program are also  available [[eman2:e2tomo_new|here]].  
 + 
 +//e2spt_refine_new.py// was integrated into e2projectmanager in mid-2022, so make sure you have an up to date EMAN2 installation.
  
 ==== Small (large below) ==== ==== Small (large below) ====
Line 452: Line 493:
  
 ===== Old Subtomogram refinement (~1 hr/iteration) ===== ===== Old Subtomogram refinement (~1 hr/iteration) =====
-{{http://blake.bcm.edu/dl/EMAN2/refinement.png|3D refinement|width="600"}} +{{https://blake.bcm.edu/dl/EMAN2/refinement.png}} 
-As an alternative to the new integrated tool above, the older pair of programs is still available. You shouldn't need to do both approaches. This step is similar to the "p" iterations above, though it uses an older algorithm.+ 
 +**This section is ONLY for use if you skipped the e2spt_refine_new steps above**. The older pair of programs is still available. This step is similar to the "p" iterations above, though it uses an older algorithm.
  
 This step performs a conventional iterative subtomogram averaging using the full set of particles. Typically it will achieve resolutions in the 15-25 A range with a reasonable number of particles. As it involves 3-D alignment of the full set of particles multiple times, it takes a significant amount of compute time. Higher resolutions are achieved in the next stage after this (subtilt refinement). This step performs a conventional iterative subtomogram averaging using the full set of particles. Typically it will achieve resolutions in the 15-25 A range with a reasonable number of particles. As it involves 3-D alignment of the full set of particles multiple times, it takes a significant amount of compute time. Higher resolutions are achieved in the next stage after this (subtilt refinement).
Line 477: Line 519:
 ===== Old Subtilt refinement (~9 hr/iteration) ===== ===== Old Subtilt refinement (~9 hr/iteration) =====
 {{http://blake.bcm.edu/dl/EMAN2/subtlt_dir.png|Subtilt refinement directory|width="600"}} {{http://blake.bcm.edu/dl/EMAN2/subtlt_dir.png|Subtilt refinement directory|width="600"}}
-This is the second half of the old refinement strategy. It is conceptually similar to the t,p and r iterations in the newer integrated program above.+**This is the second half of the old refinement strategy.** It is conceptually similar to the t,p and r iterations in the newer integrated program above.
  
 With the results of a good subtomogram alignment/average, we are now ready to switch to alignment of the individual particle images in each tilt, along with per-particle-per-tilt CTF correction and other refinements. This is effectively a hybrid of single particle analysis and subtomogram averaging, and can readily achieve subnanometer resolution IF the data is of sufficient quality. The tutorial data set is, but many cellular tomograms, for example, are not collected with high resolution in mind, and even with this sort of refinement will be unable to achieve resolutions better than 10-30 A, depending on the data. This process is completely automatic, based on all of the metadata collected up to this point. While it is possible to perform "subtomogram refinement" with subtomograms from any tomogram, Subtilt Refinement cannot operate properly unless all preceding steps occurred within EMAN2. With the results of a good subtomogram alignment/average, we are now ready to switch to alignment of the individual particle images in each tilt, along with per-particle-per-tilt CTF correction and other refinements. This is effectively a hybrid of single particle analysis and subtomogram averaging, and can readily achieve subnanometer resolution IF the data is of sufficient quality. The tutorial data set is, but many cellular tomograms, for example, are not collected with high resolution in mind, and even with this sort of refinement will be unable to achieve resolutions better than 10-30 A, depending on the data. This process is completely automatic, based on all of the metadata collected up to this point. While it is possible to perform "subtomogram refinement" with subtomograms from any tomogram, Subtilt Refinement cannot operate properly unless all preceding steps occurred within EMAN2.
Line 499: Line 541:
  
 ===== Refinement evaluation (optional) ===== ===== Refinement evaluation (optional) =====
-{{http://blake.bcm.edu/dl/EMAN2/refinement_evaluation.png|Refinement evaluation|width="600"}} This tool helps visualize and compare results from multiple subtomogram refinement runs.+{{https://blake.bcm.edu/dl/EMAN2/refinement_evaluation.png}} This tool helps visualize and compare results from multiple subtomogram refinement runs.
  
   * **Analysis and Visualization -> Evaluate SPT Refinements**   * **Analysis and Visualization -> Evaluate SPT Refinements**
eman2/e2tomosmall.1754353525.txt.gz · Last modified: by steveludtke