Protein Engineering, Vol. 13, No. 1, 15-19,
January 2000
© 2000 Oxford University Press
Is it better to combine predictions?
Department of Computer Science, University of Wales, Aberystwyth Penglais, Aberystwyth, Ceredigion, SY23 3DB, Wales, UK, 2 Department of Engineering Mathematics and Computer Science, Speed Scientific School, University of Louisville, Louisville, KY 40292, USA and 3 Department of Biostatistics and Medical Informatics and Department of Computer Sciences, University of Wisconsin, 1300 University Avenue, Room 5795 Medical Sciences, Madison, WI 53706, USA.Email: page{at}biostac.wisc.edu
 |
Abstract |
|---|
 Top Abstract IntroductionMethodsResultsDiscussionConclusionReferences |
|---|
We have compared the accuracy of the individual protein secondary structure prediction methods: PHD, DSC, NNSSP and Predator against the accuracy obtained by combing the predictions of the methods. A range of ways of combing predictions were tested: voting, biased voting, linear discrimination, neural networks and decision trees. The combined methods that involve `learning' (the non-voting methods) were trained using a set of 496 non-homologous domains; this dataset was biased as some of the secondary structure prediction methods had used them for training. We used two independent test sets to compare predictions: the first consisted of 17 non-homologous domains from CASP3 (Third Community Wide Experiment on the Critical Assessment of Techniques for Protein Structure Prediction); the second set consisted of 405 domains that were selected in the same way as the training set, and were non-homologous to each other and the training set. On both test datasets the most accurate individual method was NNSSP, then PHD, DSC and the least accurate was Predator; however, it was not possible to conclusively show a significant difference between the individual methods. Comparing the accuracy of the single methods with that obtained by combing predictions it was found that it was better to use a combination of predictions. On both test datasets it was possible to obtain a ~3% improvement in accuracy by combing predictions. In most cases the combined methods were statistically significantly better (at P = 0.05 on the CASP3 test set, and P = 0.01 on the EBI test set). On the CASP3 test dataset there was no significant difference in accuracy between any of the combined method of prediction: on the EBI test dataset, linear discrimination and neural networks significantly outperformed voting techniques. We conclude that it is better to combine predictions.
Keywords: machine-learning/neural-networks/secondary structure/statistics
|
Introduction |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionConclusionReferences |
|---|
The prediction of secondary structure is the most basic and generally applicable form of protein structure prediction (Schulz and Schirmer, 1978; King, 1996
; Lesk, 1997). Secondary structure prediction is now a standard part of fold recognition methods (e.g. Murzin and Bateman, 1997; Russel et. al., 1997), and was increasingly used in hybrid a priori fold prediction methods in CASP3 (http://PredictionCenter.llnl.gov/casp3/Casp3.html). Over the years many different approaches have been taken to predicting protein secondary structure: statistical (Robson, 1976; Garnier et al. 1978, 1996; Gibrat et al., 1987; King and Sternberg, 1996), neural network (Qian and Sejnowski, 1988; Kneller et al., 1990; Rost and Sander, 1993), nearest-neighbour (Zhang et al., 1992; Yi and Lander, 1993; Salamov and Solovyev, 1995; Frisham and Argos, 1997) and symbolic machine learning (King and Sternberg, 1990; Muggleton et al., 1992). There are now available a number of accurate prediction methods freely available over the internet. The best of these methods are estimated to have accuracies of ~70% (per residue): PHD (Rost and Sander, 1993; http://www.embl-heidelberg.de/predictprotein/ppDoPredDef.html), DSC (King and Sternberg, 1996; http://www.icnet.uk/bmm/dsc/dsc_form_align.html), NNSSP (Salamov and Solovyev, 1995; http://genomic.sanger.ac.uk/pss/pss.shtml) and Predator (Frisham and Argos, 1997; http://www.embl-heidelberg.de/argos/predator/run_predator.html). In this paper we seek to answer the following questions. Is it better to use a single secondary structure prediction method or to combine predictions from several different methods? If it is better to combine predictions, what is the best method for doing so?
Basic probability theory states that all of the evidence relevant to a prediction should be used in making that prediction (Jaynes, 1994). This suggests that if a novel prediction method provides new relevant information, then it should be possible to combine this with existing methods to obtain improved predictions. This idea is the basis of ensemble learning and multi-strategy learning methods, which are currently important subjects in machine learning (Dietterich, 1997). The standard approach in multi-strategy learning is to identify different prediction or learning tasks within one problem-solving activity, and independently apply the most appropriate strategy to each task (Tecuci, 1998; Califf and Mooney, 1999). Ensemble learning, on the other hand, focuses on a single prediction task, and builds multiple predictors or classifiers (an ensemble) for that task. The different predictors are combined either by voting or by training a classifier to combine them. In the majority of ensemble learning applications, the same machine learning algorithm is used, and only the training data is changed. This is what is done in the popular forms of ensemble learning known as bagging and boosting: in bagging, different prediction methods are repeatedly learnt using different bootstrap samples (Breiman, 1996); in boosting, the sample is `reweighted' to give more importance to cases that are incorrectly predicted (Freund and Shapire, 1996). Bauer and Kohavi (1999) provide an empirical comparative analysis of boosting, bagging and related ensemble approaches.
Within the field of protein secondary structure prediction the idea of combining different prediction methods is also an old one. Schultz et al. (1974) combined 11 methods, Argos et al. (1976) combined five. Sternberg (1983) stressed the merits and limitations of different prediction methods for different types of secondary structure (multi-strategy learning). Nishikawa and Ooi (1986) combined methods using a voting strategy. Biou et al. (1988) used a complicated biased method to combine three methods (including GOR 111). Zhang et al. (1992) used a neural network (which they found to be better than voting) to combine predictors obtained from neural network, nearest neighbour and naive Bayes algorithms. Rost and Sander (1993) in PHD learnt several different neural networks and averaged their output to produce a final prediction. Salamov and Solovyev (1995) used majority voting to combine different nearest neighbour predictors. King and Sternberg (1996) showed that combining their program DSC with the neural network method PHD produced an improved prediction accuracy. Recently, Cuff and Barton (1999) carried out an extensive comparison of different prediction methods: DSC v 1.0, PHD, NNSSP, Predator, ZPRED (Zvelebil et al., 1987) and MULPRED (G.J.Barton, unpublished). They also set up a server for these different prediction methods: http://circinus.ebi.ac.uk:8081/. The authors tried using weighted votes and neural networks to combine methods, but concluded that a simple majority vote of DSC v 0.1, PHD, NNSSP, Predator (with PHD being used to break ties) was the best.
A central problem with work on combing protein secondary structure prediction methods is obtaining an accurate and unbiased estimate of the accuracy of the combined method. This is because authors of combined methods typically have not had access to an unbiased test set. For example, in King and Sternberg (1996) the combined results were biased optimistically because the PHD predictions were based on protein domains that PHD had trained on; a similar problem faced Cuff and Barton (1999) whose results were also optimistically biased because their test data included protein domains that NNSSP and Predator have been trained on.
To obtain an unbiased estimate it is necessary to test on data that none of the individual methods in the combined method have been trained on. There are two possible ways to do this:
- (i)Retrain all the methods (preferably using leave-one-out cross-validation) on a common set of sequences. This would give complete control over the training and ensure that the evaluation of the combined methods is completely fair. This would be the ideal solution, but it is not practical because it is not possible to obtain the code to retrain all the methods.
(ii)To find a new set of proteins of known structure that has not been seen by any of the existing methods and use this for testing. The use of a `clean' testing set would allow a statistically unbiased prediction of the accuracy of a combined prediction method. The combination of prediction methods that performs best on this new test data is the one that is most likely to perform well in the future. If a combined prediction performs significantly better that any single method then it is highly likely that this combination will better on any new proteins.
In this paper we use the second approach of testing prediction methods on unseen datasets. We use two unseen datasets, one from CASP3 (the Third Community Wide Experiment on the Critical Assessment of Techniques for Protein Structure Prediction) and one from the European Bioinformatics Institute.
|
Methods |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionConclusionReferences |
|---|
Data
Training data
The training data used was a set of 496 non-homologous domains. This dataset is based on the one developed by Cuff and Barton (1999) and it is almost a proper superset of a training set of 126 domains used to originally train PHD (Rost and Sander, 1993) and DSC (King and Sternberg, 1996). The reason that it is not quite a superset is that the definition of homology used is now stricter). The dataset differs slightly from that of Barton in excluding proteins with less than 30 residues. There are 82 847 residues in this dataset: 28 678 in 
-helix conformation, 17 741 in ß-strand and 36 428 in coil. Secondary structure was assigned using DSSP (Kabsch and Sander, 1983). Cuff and Barton (1999) have shown that the exact mapping of DSSP output to three state secondary structure can have a significant effect on the resulting estimated accuracy. We have used the conservative mapping: H, I, G are translated to -helix; E is translated to ß-strand, and all other states are translated as coil. This dataset is based on a 1996 release of PDB.
Test data
We used two separate test sets. The first test set consisted of 17 domains which the CASP3 organizers identified as having no known homologous structures in PDB (http://PredictionCenter.llnl.gov/); and hence no homology with our training set. We used the 17 domains which were chronologically last to be revealed before the CASP3 meeting; we could not use any more CASP3 proteins as we did not start the experiment in time. These proteins are given in Table I. The data consists of 2373 residues, 886 in -helix conformation, 433 in ß-strand and 1054 in coil. Residues not seen in the protein structures were assigned as coil. This dataset is somewhat small to reveal significant statistical differences, but there was no possibility that any prediction method could have seen this dataset as all the predictions of the individual methods were carried out before the structures were revealed.
|
The second dataset considered was generated by James Cuff and Geoff Barton at the European Bioinformatics Institute (EBI) using the same procedure as the training data but on an updated release of PDB. The dataset consists of all non-homologous domains added to PDB since formation of the training set in 1996. This dataset consists of 405 domain. There are 81 911 residues, 28 277 in
-helix conformation, 18 591 in ß-strand and 35 043 in coil. This dataset is large enough to identify differences between prediction methods, but it is possible that a few domains were used to train one or more of the prediction methods. If the same broad results are obtained on both datasets then these results would have strong experimental support.
Alignments We formed alignments for all the proteins using the basic method described for DSC in King et al. (1997); we used the nr database instead of OWL, it contained 90 372 249 letters and 299 576 sequences and it was posted on April 17, 1998.
Algorithms
Secondary structure prediction algorithms
In CASP2 (http://PredictionCenter.llnl.gov/casp2/Casp2.html) three publicly available methods clearly performed better than any others (Lesk, 1997). These methods were PHD, DSC and NNSSP. In the two years since CASP2 the program Predator (Frisham and Argos, 1997) has also appeared and become popular. We therefore chose to investigate the use of PHD, DSC, NNSSP and Predator in our analysis.
PHD (Rost and Sander, 1993) is an accurate prediction method and probably the most commonly used secondary structure prediction method. It has a publicly available server (http://www.embl-heidelberg.de/predictprotein/ppDoPredDef.html). PHD was the most accurate method in both CASP1 (Defay and Cohen, 1995) and CASP2 (Lesk, 1997). In PHD multiple alignments are represented by a window of normalized sequence frequencies. The learning method is the neural network algorithm back-propagation, using both sequential and combined nets with single hidden layers.
DSC (King and Sternberg, 1996) is an accurate predic- tion method that has both a public server (King et al., 1997; http://www.icnet.uk/bmm/dsc/dsc_form_align.html) and publicly distributed code (ftp://ftp.icnet.uk/pub/bmm/king/dsc/dsc.tar.gz). In DSC, sequences are described using a set of sequence descriptors (residue conformation propensities, sequence edge effects, moments of hydrophobicity, position of insertions and deletions, moments of conservation, auto-correlation, residue ratios, secondary structure feedback effects and filtering); multiple sequences in an alignment are used in a simplistic way. The learning method is standard linear discrimination. We used DSC v1.2–this version includes an updated and more accurate GOR 1 matrix. This has been found to improve prediction accuracy by 0.5–1%. The matrix was derived from a dataset of 496 proteins compared with the original dataset of 126.
The NNSSP prediction method (Salamov and Solovyev, 1995) is a successful prediction method with its own server (http://genomic.sanger.ac.uk/pss/pss.shtml). Sequences are represented as multiple alignments and information about insertions and deletions used. The learning method of NNSSP is (50 and 100)-nearest neighbours with a voting system to combine predictions. We carried out the NNSSP predictions using the binary code given to us by Dr Solovyev and the clu2nssp program written by Nicolas Le Novère to convert the clustalw format to nnssp format.
The Predator prediction method (Frisham and Argos, 1997) is the most recent prediction method considered in this paper. It has its own prediction web server (http://www.embl-heidelberg.de/argos/predator/run_predator.html) and the code is available (ftp://ftp.ebi.ac.uk/pub/software/unix/predator). Predator is based on a combined nearest neighbour and statistical approach that aims to recognize hydrogen bonding patterns. We carried out the Predator predictions using the downloaded code with our own alignments.
Combined dataset
Using the predictions of the different methods we formed, using the CASP3 test protein, a dataset containing the attributes described in Table II with the class value (value to be predicted) either -helix, ß-strand or coil.
|
Methods used for combing predictions We investigated the following methods for combing predictions: voting, biased voting, linear discrimination, neural networks and decision trees. We choose these methods as they are simple, generally used and successful (Weiss and Kulikowski, 1991
; King et al., 1995).
- In the `simple vote' we took a majority vote of the predictions of PHD, DSC v1.2, NNSSP and Predator. When there was no majority, we chose to predict a coil state (as it is the most common state). Voting is known to be a robust strategy and can under certain assumptions be shown to approach optimality (Mitchell, 1997).
- In the `biased vote' we used the predictions of PHD, DSC v1.2, NNSSP and Predator, but only considered residues where PHD and DSC v1.2 disagreed. For these predictions, if PHD agreed with either NNSSP or Predator we choose PHD, else if DSC agreed with either NNSSP or Predator we chose DSC, else if NNSSP and Predator agree then we chose NNSSP, else if all disagree then we chose PHD. This biased voting scheme exploits our prior belief that PHD was probably the most accurate method of the four, and that DSC is less overfitted on our training data than NNSSP or Predator (because it is based on linear discrimination).
- Linear discrimination is one of the oldest, most basic and successful of all statistical learning methods (Fisher, 1936; Weiss and Kulikowski, 1991; King et al., 1995). It is also known to be based on robust assumptions (Jaynes, 1994). We used our own implementation of linear discrimination for learning.
- For neural networks, we used the standard back-propagation algorithm to train a network to decide on a combined prediction using the popular SNNS package (ftp://ftp.informatik.uni-stuttgart.de/pub/SNNS/). The attributes were the same as in linear discrimination. The problem was represented for artificial neural networks such that the input parameters were mapped as 11 to an input layer feeding a single hidden layer with six neurons (Table II
). The output layer provides three outputs representing C, E and H. The neural network selected was based on back-propagation using momentum with the coding of the variables , ß and c for the outputs as 100, 010 and 001. It was decided to use a `winner takes all' concept by selecting the highest output value of the three possible outputs to be mapped to a one. This removed ambiguity from border-line decision cases by thresholding 0.8, 0.6, 0.2 to become 1, 0, 0, which is essentially the representation of C. Without such a thresholding decision it would have been impractical to make choices.
- For decision tree we used the C5 program (http://www.rulequest.com/). C5 is a popular and commonly used propositional algorithm that learns decision trees and rules; it is the success to the successful C4.5 (Quinlan, 1993). We used default settings for all values. C5 performed so poorly on the CASP3 test set one that we did not apply it to the second dataset.
Statistical tests We used a binomial sign test to compare if one prediction method is better. The procedure for the binomial sign test is to compare the accuracy of two methods on each protein in the test data and count how many times one method was more accurate than another. The probability of obtaining these results assuming that there was no difference between the methods was then calculated using the binomial distribution (one-tailed as we are testing if the more accurate method really was more accurate). The binomial sign test makes no assumptions about the distribution of accuracies of the methods, but does not use the size of differences in accuracy (i.e. 0.1% more accurate is considered the same as 10% more accurate).
|
Results |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionConclusionReferences |
|---|
Results on CASP3 test dataset
The results of the basic prediction methods PHD, DSC v1.2, NNSSP and Predator on the CASP3 test data are given in Table III. The order of accuracy from most accurate to least was NNSSP, PHD, DSC, Predator. However, there was no significant difference between the accuracy of any of the methods at P = 0.05. The results of testing the different combined methods on the CASP3 test data are also given in Table III. In this database, attributes derived from PHD, DSC v1.2, NNSSP and Predator (Table II) were combined to directly predict the secondary structure class. The combined methods are from 1–3% more accurate. The biased vote method was found to be the most accurate and C5 the least accurate. There were no significant differences between combining methods except that C5 was significantly worse than using voting, linear discrimination and neural networks at P = 0.01. The combined results were also compared with that of the individual methods. All the combined methods are more accurate, with the difference generally significant at P = 0.05. The evidence is clearly supports the conclusion that it is better to combine predictions.
|
Results on Cuff and Barton test dataset
The results of the basic prediction methods PHD, DSC v1.2, NNSSP and Predator on the CASP3 test data are given in Table IV. The order of accuracy is the same as for CASP3: NNSSP, PHD, DSC, Predator. However, because a small number of proteins may have been seen by some methods, it is not justified to say that one method is significantly more accurate than any other. However, this possibility can make no difference to the comparison of single and combined methods. The results of testing the different combined methods on the CASP3 test data are also given in Table IV. The combined methods are from 1–3% more accurate. The results show that the more sophisticated methods of combining predictions–linear discrimination and neural networks–are significantly more accurate (at P = 0.01) than the voting methods for combining predictions. This supports the result of Zhang et al. (1992) who successfully combined predictions using neural networks, and contradicts the results of Cuff and Barton (1999) who could not get an improvement using neural networks. A possible explanation for this disagreement is that they used directly the predictions obtained from PHD, DSC, NNSSP and Predator (i.e. , ß, c recoded as binary are taken as input by their neural network), while we used the probabilities of each state and also the number of classifiers predicting -helix or ß-strand. It seems that their strategy leads to the loss of some information. The authors themselves suggested that better accuracies would be achievable using the propensities of the different states, rather than simple binary input.
|
The combined results were also compared with that of the individual methods. In all comparisons (except simple vote via NNSSP) the combined methods were significantly more accurate (at P = 0.01). The clear conclusion is that it is better to combine predictions.
|
Discussion |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionConclusionReferences |
|---|
The prediction servers for PHD, DSC, NNSSP and Predator will, sooner or later, become outdated and redundant. In CASP3 (December 1998) these methods were no longer the most accurate (http://PredictionCenter.llnl.gov/casp3/Casp3.html), although they performed competitively. The question therefore naturally arises whether the results we have obtained about how best to combine prediction methods will be capable of being carried forward to new secondary structure prediction methods, and more generally, whether our results have application to other prediction problems. The following recommendations are based on general statistical principles and are in good agreement with our results.
It is better to combine prediction methods (assuming that they do not differ greatly in accuracy).
- Voting is a robust way to combine prediction methods when you are unsure if your training data is based on over-fitted predictions.
- You should use prior knowledge about the strengths of prediction methods to bias voting.
- It may be possible to improve on voting by using learning methods, but care is needed and results checked on independent test data.
|
Conclusion |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionConclusionReferences |
|---|
In this paper we have compared the accurate and commonly used protein secondary structure prediction methods–PHD, DSC, NNSSP and Predator–against the accuracy of combining the predictions of all four methods. A range of different ways of combing the predictions were tried: voting, biased voting, linear discrimination, neural networks and decision trees. We used two separate test sets and the results were qualitatively the same on both: it is better to combine predictions than to use a single method.
|
Acknowledgments |
|---|
Ross D.King and Mohammed Ouali were funded by the BBSRC/EPSRC Bioinformatics initiate grant BIF08765. We would like to thank the organizers of CASP3 for collecting the new crystal structures and all the crystallographers who donated structures to CASP3. We would also like to thank James Cuff and Geoff Barton of the EBI for kindly allowing us to use their datasets.
|
Notes |
|---|
1 To whom correspondence should be addressed
|
References |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionConclusionReferences |
|---|
Argos,P., Schwartz,J. and Schwartz,J. (1976) Biochim. Biophys. Acta, 439, 261–273.[Medline]
Bauer,E. and Kohavi,R. (1999) Machine Learning, in press.
Biou,V., Gibrat,J.F., Levin,J.M., Robson,B. and Garnier,J. (1988) Protein Engng, 2, 185–191
Breiman,L. (1996) Machine Learning, 24, 123–140.
Califf,M. and Mooney,R. (1999) In Proceedings of the Sixteenth National Conference on Artificial Intelligence (AAAI-99), in press.
Cuff,J.A. and Barton,G.J. (1999) Prot. Struct. Funct. Genet., 34, 508–519.
Defay,T. and Cohen,F.E. (1995) Prot. Struct. Funct. Genet., 23, 431–445.
Dietterich,T.G. (1997) AI Magazine, 18, 97–136.
Fisher,R.A. (1936) Eugenics, 7, 179–188.
Freund,Y. and Shapire,R.E. (1996) In Proceedings of the Thirteenth International Conference on Machine Learning. Morgan Kaufmann, San Francisco, pp. 148–156.
Frishman,D. and Argos,P. (1997) Proteins, 27, 329–335.[Web of Science][Medline]
Garnier,J., Osguthorpe,D.J. and Robson,B. (1978) J. Mol. Biol., 120, 97–120.[Web of Science][Medline]
Garnier,J., Gibrat,J.F. and Robson,B. (1996) Methods Enzymol., 226, 540–553.
Gibrat,J.F., Garnier,J. and Robson,B. (1987) J. Mol. Biol., 198, 425–443.[Web of Science][Medline]
Jaynes,E.T. (1994) Probability Theory: The Logic of Science. http://omega.albany.edu:8008/JaynesBook.html
Kabsch,W. and Sander,C. (1983) Biopolymers, 22, 2577–2637.[Web of Science][Medline]
King,R.D. (1996) In Sternberg,M.J.E. (ed.), Protein Structure Prediction: A Practical Approach. Oxford University Press, Oxford, pp. 79–97.
King,R.D. and Sternberg,M.J.E. (1996) Protein Sci., 5, 2298–2310.[Web of Science][Medline]
King,R.D. and Sternberg,M.J.E. (1990) J. Mol. Biol., 216, 441–457.[Web of Science][Medline]
King,R.D., Feng,C. and Sutherland,A. (1995) Appl. Artificial Intelligence, 9, 289–335.
King,R.D., Saqi,M., Sayle,R. and Sterenberg,M.J.E. (1997) CABIOS, 13, 473–474.
Kneller,D.G., Cohen,F.E. and Langridge,R. (1990) J. Mol. Biol., 214, 171–182[Web of Science][Medline]
Lesk,A.M., (1997) Prot. Struct. Funct. Genet., (Suppl. 1), 151–166.
Mitchell,T.M. (1997) Machine Learning. McGraw Hill, New York.
Muggleton,S., King,R.D. and Sternberg,M.J.E. (1992) Protein Engng, 5, 647–657.
Murzin,A.G. and Bateman,A. (1997) Prot. Struct. Funct. Genet., (Suppl. 1), 105–112.
Nishikawa,K. and Ooi,T. (1986) Biochem. Biophys. Acta, 871, 45–54.[Medline]
Qian,N. and Sejnowsk,T.J. (1988) J. Mol. Biol., 202, 865–884.[Web of Science][Medline]
Quinlan,J.R. (1993) C4.5: Programs for Machine Learning. Morgan Kaufmann, San Mateo.
Robson,B. (1976) J. Mol. Biol., 107, 327–356.[Web of Science][Medline]
Rost,B. and Sander,C. (1993) J. Mol. Biol., 232, 584–599.[Web of Science][Medline]
Russell,R.B., Saqi,M.A., Sayle,R.A., Bates,P.A. and Sternberg,M.J.E. (1997) J. Mol. Biol., 269, 423–439.[Web of Science][Medline]
Salamov,A.A. and Solovyev,V.V. (1995) J. Mol. Biol., 247, 11–15.[Web of Science][Medline]
Schultz,G.E. and Schirmer,R.H. (1978) Principles of Protein Structure. Springer-Verlag, Berlin.
Schultz,G.E. et al. (1974) Nature, 250, 140–142.[Medline]
Sternberg,M.J.E. (1983) In Geisow and Barrett (eds), Computing in Biological Science. Elsevier Biomedical Press.
Weiss,S.M. and Kulikowski,C.A. (1991) Computer Systems that Learn. Morgan Kaufmann, San Mateo.
Yi,T. and Lander,E.S. (1993) J. Mol. Biol., 232, 1117–1129.[Web of Science][Medline]
Zhang,X., Mesirov,J.P. and Waltz,D.L. (1992) J. Mol. Biol., 225, 1049–1063.[Web of Science][Medline]
Zvelebil,M.J.J.M., Barton,G.J., Taylor,W.R. and Sternberg,M.J.E. (1987) J. Mol. Biol., 195, 957–961.[Web of Science][Medline]
Received April 28, 1999; revised October 21, 1999; accepted October 27, 1999.
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  M. Albrecht, S. C.E. Tosatto, T. Lengauer, and G. Valle Simple consensus procedures are effective and sufficient in secondary structure prediction Protein Eng. Des. Sel., July 1, 2003; 16(7): 459 - 462. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
C. L. Wilson, S. J. Hubbard, and A. J. Doig A critical assessment of the secondary structure {alpha}-helices and their termini in proteins Protein Eng. Des. Sel., July 1, 2002; 15(7): 545 - 554. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 D. B. Kell, R. M. Darby, and J. Draper Genomic Computing. Explanatory Analysis of Plant Expression Profiling Data Using Machine Learning Plant Physiology, July 1, 2001; 126(3): 943 - 951. [Full Text] [PDF]
|
|
|
|
|
|
A. C.R. Martin The ups and downs of protein topology; rapid comparison of protein structure Protein Eng. Des. Sel., December 1, 2000; 13(12): 829 - 837. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 R. Gijsbers, H. Ceulemans, W. Stalmans, and M. Bollen Structural and Catalytic Similarities between Nucleotide Pyrophosphatases/Phosphodiesterases and Alkaline Phosphatases J. Biol. Chem., January 5, 2001; 276(2): 1361 - 1368. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||