Table 4. Effect of sample size on estimates of lexical specificity in Lara’s wh-questions





Caroline F. Rowland, Sarah L. Fletcher and Daniel Freudenthal



those that are most likely to occur in the early samples. For example, suppose that, in

order to investigate the order of acquisition of different verbs, we collect 100 utterances per week for five weeks. We are very likely to capture frequent verbs (e.g., verbs

that occur at least once per 100 utterances) in our very first sample (after we have collected 100 utterances). However, verbs that occur less frequently are very unlikely to

occur in our first sample. For example, it is only after 2 weeks (i.e. after we have collected 200 samples) that we are likely to capture at least one example of verbs that occur once every 200 utterances. It will take us five weeks (500 utterances) before we can

be certain of capturing a verb that occurs once every 500 utterances. In other words,

more frequent verbs are more likely to occur in earlier samples (and thus be identified

as early acquired) than less frequent verbs, even if both verbs were acquired before the

beginning of the sampling period.



3.3



The effect of vocabulary size on productivity measures



The third possible confound on estimates of productivity is the fact that children’s vocabularies are smaller than those of adults. Since speakers can only produce utterances

using vocabulary items they have already learned, children are less likely than adults to

be capable of demonstrating productivity with a wide range of grammatical structures.

For example, a child who knows only two determiners will have far less opportunity to

demonstrate a sophisticated knowledge of the determiner category than a child who

knows four, even if both children have equally abstract knowledge of the category

(Pine and Lieven 1997). Thus, lexical specificity in the data could also be due to a limi­

ted knowledge of vocabulary, not to limited grammatical knowledge.



3.4



Assessing productivity: A solution



To recap, the accuracy with which any one sample assesses productivity is affected by

sample size, by the frequency statistics of the language, and by the vocabulary size of

the child. Importantly, even collecting much bigger samples will not overcome these

problems. There will still be an impact of sample size and frequency statistics on measures of productivity, no matter how many utterances are collected. In addition, children’s limited vocabulary knowledge will still affect the range and variability of the

syntactic structures they produce. In order to attribute limited productivity to children

reliably it is important to control for the effect of sample size and vocabulary, while

taking into account the frequency statistics of the language. The best way to do this is

to use a comparison measure based on a matched sample of adult data.

Aguado-Orea and Pine’s (Aguado-Orea and Pine 2002; Aguado-Orea 2004) study

on Spanish verb morphology provides such a comparison measure. They investigated

the productivity of children’s verb morphology in Spanish, controlling for a number of

methodological factors that could explain limited flexibility in verb inflection use.







How big is big enough 



­ sing the dense data from the two children discussed in section 2.2.3 (Juan and Lucia,

U

aged 2;0 to 2;6), they investigated the effects of (a) limited vocabulary, (b) limited sample size, and (c) limited knowledge of particular inflections, on estimates of productivity. They reasoned that if limited productivity in children’s speech was due to these

three methodological constraints, there should be no difference between estimates of

productivity based on children’s and adults’ speech, if the samples were matched on

vocabulary, sample size and knowledge of inflections. However, if the children’s speech

was significantly more limited than we would expect, given the size of their samples

and their knowledge of verbs and inflections, we should find significant differences

between estimates of productivity based on child and adult speech.

The study focused on present tense verb inflectional contexts and the measure of

productivity used was the average number of inflections per verb (where one inflection per verb was the minimum level of productivity and four inflections per verb was

the maximum).9 The analyses compared the child’s speech with that of his or her own

primary caregivers (mothers and fathers).

The researchers controlled for knowledge of inflection by restricting the analysis

to those transcripts recorded after the child had already produced the inflections in his

or her speech. They controlled for vocabulary by restricting the analysis to verb stems

that occurred in both the child’s and the adult’s speech. Finally, they controlled for

sample size by excluding a random number of utterances from the larger of the two

samples, so that both samples contained the same number of verb tokens. Table 5 provides a summary of results, based on data from Aguado-Orea (2004).

The results for the adults clearly demonstrated that restricting sample size, vocabulary and inflection knowledge had an impact on the extent to which the speakers

were able to demonstrate productivity. Presumably all four adults had productive

knowledge of all four inflections and how to apply them to all verbs, but they only

produced between 2.48 and 2.17 inflections per verb in the samples. Similarly, neither

Juan nor Lucia was able to show knowledge of more than 2.24 inflections per verb,

even in the biggest samples. However, importantly, the children’s use of verb inflection

was always significantly less productive than that of their mothers and fathers, and

improved over time to more adult like levels. Thus, although there was a substantial

effect of limited lexical knowledge and of sample size, Aguado-Orea and Pine demonstrated that it is possible to find evidence for limited productivity in children by comparing adult and child data in order to control for these confounds.



9. The requirement to control for knowledge of inflection restricted the analysis to only four

of the six present tense inflections because two were produced too late on in the collection process to yield enough data. The inflections finally included were 1st singular, 2nd singular, 3rd

singular and 3rd plural inflections.







Caroline F. Rowland, Sarah L. Fletcher and Daniel Freudenthal



Table 5.  Average number of inflections per verb in the data from Juan, Lucia and their parents.

Participant



No. of verb tokens



No. inflections per verb



Juan (sample equivalent to father)

Juan’s father



2414

2414



2.18

2.44



Juan (sample equivalent to mother)

Juan’s mother



2058

2058



2.24

2.35



Lucia (sample equivalent to father)

Lucia’s father



874

874



1.87

2.48



Lucia (sample equivalent to mother)

Lucia’s mother



809

809



1.90

2.17



To conclude this section, the effects of sample size, frequency statistics and vocabulary

limitations on children’s utterances are large. Adults are able to demonstrate a much

greater degree of productivity in their speech than children, simply because they speak

more – yielding bigger samples of speech for analysis – and because they possess a

larger vocabulary – allowing them to demonstrate their grammatical knowledge with

a wider range of words and a larger number of structures. When samples of adult

speech are equated to samples of child speech on these measures, the apparent productivity of adult speech reduces substantially.

However, it remains possible to demonstrate lexical specificity in child’s speech,

even when the appropriate controls are applied. Aguado-Orea and Pine (2005) demonstrated that Spanish children produced significantly fewer inflections per verb than

adults, even after the application of methodological controls. Pine and Martindale

(1996), in a study of determiner acquisition, reported similar findings: applying controls for vocabulary and sample size reduced the difference between the productivity

of child and adult speech, but, children’s utterances remained significantly more lexically-specific than those of adults. Rowland and Fletcher (2006) showed that Lara’s

wh-question use was more restricted than a matched sample of maternal questions,

once knowledge of wh-word and auxiliary and sample size was equated. However, the

difference between the composition of adult and of child speech is likely to be less

striking than has sometimes previously been claimed.



4. Conclusion

In the present chapter, we have demonstrated some of the possible consequences of

taking sampled naturalistic data at face value. First, we have shown that estimates of

error rates calculated using small samples of data may be misleading, either over- or

under-estimating error rates quite substantially or even failing to capture rare errors







How big is big enough 



altogether. Second, we have illustrated that analyses of error must incorporate the fact

that error rates are likely to change over time and that errors may be more frequent in

some parts of the system than in others. Analyses of overall error rates (collapsed

across time or across sub-systems) will disproportionately reflect how well children

perform with high frequency items or how well children are doing at the later stages of

development (when children tend to produce more utterances). Since errors seem to

be more frequent at earlier points of development and in low frequency structures,

overall error rates are likely to under-estimate error rates in low frequency structures.

One solution to the sampling problem lies in suiting the sampling regime to the

structure under investigation – whether by mathematical methods such as hit probability, or by using different sampling techniques. Another solution lies in calculating

average error rates across a number of samples – whether across children or across

different samples from the same child. Although averaging error rates across children

will give no indication of the scale of the impact of individual differences or of different

sampling densities, inspection of the range and standard deviation, as well as the mean

error rate, will give researchers an indication of the heterogeneity of the samples and

allow further investigation if there is evidence for substantial variation.

Second, we have demonstrated that estimates of productivity are affected by the

sampling regime in three ways. First, in spoken languages, a small number of highly

frequency words dominate utterances, so apparent limited productivity may simply reflect the frequency statistics of the language being spoken. Second, the greater the sample

size, the more utterances will be collected and the more productive the speaker will appear. Since children tend to produce fewer utterances per minute than adults (at least

early in the acquisition process), children’s utterances are bound to seem less productive.

Third, a child who knows only a small number of words will be unable to demonstrate

the same level of productivity as an adult. We have shown that with small sample sizes,

even adults can appear to demonstrate limited productivity, but that it is possible to investigate the development of productivity in child speech, while controlling for sampling

and vocabulary constraints, by comparing matched samples of adult and child data.

Given the constraints imposed by sampling on naturalistic data analysis, one

might argue that we should abandon the use of naturalistic data in favour of experimental techniques. We would argue that this is too extreme a reaction to the constraints. At the very least, the analysis of naturalistic data allows us to identify phenomena that we can then investigate further in an experimental context. However, we

suggest that the analysis of naturalistic data can provide more than just the initial description of a phenomenon. Naturalistic data analysis avoids some of the pitfalls of

experimental techniques (e.g., the Clever Hans effect) and can reveal levels of sophistication in children’s behaviour that are simply not captured in an experimental situation (see, for example, Dunn’s (1988) work on the development of social cognition). It

is important, though, to apply controls, as we would to experimental techniques, and

to take account of the confounds inherent in using naturalistic data to interpret and

evaluate theories of language acquisition.