Possibilities for evaluation of the pilot phase of the Learning Hubs project - this is an extract from a draft document provided by Roddy Stuart, external consultant.

There are several possible frameworks to use as a basis for monitoring and assessment of success throughout the 2 year pilot phase. They would include the following:

1 A curricular framework which is aligned with the emerging new curriculum for all young Scots aged 3 to 18, under the increasingly familiar 4 capacities of learners as
• successful learners
• effective contributors
• confident individuals
• responsible citizens

2 An overlapping curricular framework relates to the design principles articulated in A Curriculum for Excellence, which looks for the following experiences for all young Scots
• challenges and enjoyment
• breadth
• progression
• depth
• personalisation and choice
• coherence
• relevance

3 A framework closely associated with the published and agreed goals of the project, including these
• integration of new technology in learning and teaching in ways to offer a more personalised learning experience and with evidence of pedagogic change
• an expansion of the options facing young people in the schools concerned about what to learn and how to learn
• greater engagement with the learning process for all, especially those most at risk of graduating into the ranks of the NEET group of young people
• contextualisation of technology-based learning to reflect the real world's issues and practice
• bridging the gap between home and school especially in terms of ICT use, and including reducing the growing digital divide between 'haves' and 'have nots'

4 Less likely to apply is a framework which relates to the issues which have to be faced by the pilot schools and their managing EAs, which was developed in a paper for the schools which was written initially by this consultant in March 2007 and which forms part of the project wiki - http://121.wikispaces.com/QuestionsForSchools

Questions for the professionals
This set might apply to class teachers (primary) and subject teachers (secondary), to school managers and to EA officers

Part A: curricular ambitions
NB For this set, I would recommend use of a simple 4 point scale as follows. This would allow a straightforward comparison between the 'before' in Autumn 2007 and an 'after' in 2009
1 Regression - the project will likely get in the way
2 No evidence of progress is expected
3 Expectations of modest progress - some of the learners may exhibit such behaviour some of the time
4 Expectations of real progress - most of the learners will exhibit such behaviour commonly

In my experience, having used this approach several times, it provides a starting point for the interviewee to adduce evidence from their experience for their judgement, and it is this nuance which is the real goal of the interview
1 Enthusiasm (associated with successful learners)
2 High standards of work (also associated with successful learners)
3 Openness (again associated with successful learners)
4 Enterprise (associated with effective contributors)
5 Self-reliance and resilience (also associated with effective contributors)
6 Self-respect and emotional wellbeing (associated with confident individuals)
7 Ambition (also associated with confident individuals)
8 Participation and working with others (associated with responsible citizens)

Part B: operational issues
This set would be fairly straightforward questions about their early experiences and their advice about aspects of the pilot which need attention
1 Technical matters which may have been inhibiting
2 Clarity of purpose
3 Appropriate forms of support for the staff involved in the pilot

Part C: philosophical issues

The first set of questions might be put to all; but the second might be used only for Castlebrae High

1 How should we judge progress in the following important but less than concrete aspects of learner behaviour?
• The benefits deriving from one-to-one operation versus the clearer set of goals associated with their use
• Being enterprising
• Being ambitious
• Being engaged

2 What additional measures might be appropriate - on top of looking at attendance patterns and attainment performance - in judging success for the 'potential NEET' group?

Questions for the young people

These interviews should be conducted in groups of 4 to 6, to provide a balance between collective and individual (idiosyncratic) responses within the limited time available

1 You are now making use some of the time of a Nokia N800 tablet/Vario II device. I want you to think back over the last couple of weeks and tell me a little about a recent experience you had using these for
  • getting resources from the internet/web
  • communicating with other people
  • playing media content
2 Give me 2 adjectives or short phrases which describe your view of these devices
3 Have you had any experience of using these outside school hours yet?
4 Have you had any frustrations when using the new devices?
5 Do you think your teachers are as keen on these as you are?
6 Using the scale 1 to 4 on the separate sheet*, do you expect these to help you with the following features of your learning?
  • Being more interested in your lessons?
  • Being able to produce better work?
  • Being more willing to try out new things?
7 Can you put into words how these might help you to learn better?
8 Can you suggest how we should all judge the success of the project?

The scale for question 6 would be as follows, in large print on a couple of sheets visible to the group
  • 1 Harmful - using the devices will just waste time
  • 2 No difference - I don't think it will make any difference one way or the other
  • 3 Some improvement - I think it might help a wee bit
  • 4 Big improvement - I think that it will help a lot

I would suggest that trying to develop questions for the other stakeholders at this stage - parents and any partners - would not be worth the considerable effort

Possible sample
This is not a statistical exercise and so there is no need to look for large numbers. The following sample might be worth using
• the 3 P6 teachers in Sciennes Primary
• perhaps a cross-section of 4 of the subject teachers involved in Castlebrae High
• a school manager from Sciennes Primary
• a school manager from Castlebrae High
• there is a case for adding a 'mover' (who may not fit the above categories) in each of the schools; but they will have other ways of making their points and so it is not a clear-cut judgement; and certainly not if these were to conduct some or all of the interviews
• someone from City of Edinburgh Council who has a broad overview of the project
• a group (4-6) of children from each of the 3 P6 classes in Sciennes
• 2 or perhaps 3 groups of 4 or 5 students who are involved in Castlebrae High (although this represents a considerably greater proportion of the cohort than in Sciennes Primary)

Further interim methods
There is a range of tools which might be useful to help to sustain the sort of framework for judgement of progress which is implicit in this proposal, and can be undertaken in the development phase of the pilot (probably January to June 2008) and in the mature phase of the project (probably September 2008 to March 2009). Frequency would obviously depend on what else is going on
• Use the existing wiki (or new pages) to encourage the young people and the staff and their parents and other stakeholders to record their great successes, their bad experiences
• Peer review between staff, and perhaps in a more modest way among the younger participants; this might be associated with agreed milestones in the pilot phases, but probably not exceeding 2 or 3 times in total
• Formal logs or diaries completed by staff and perhaps younger folk over a set period, perhaps 2 or 3 weeks, again probably not exceeding 2 or 3 times in total
• Production of a podcast by the participants, perhaps starting around March 2008, to reflect their own 'take' on the project and with control over content - with negotiated guidelines about taste and privacy (and copyright)
• Production of a video record - still images and moving images, with the production of music to accompany where appropriate