1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.
  2. Hi Guest, welcome to the TES Community!

    Connect with like-minded professionals and have your say on the issues that matter to you.

    Don't forget to look at the how to guide.

    Dismiss Notice

Author Consultation Follow-up 1: Defining quality

Discussion in 'TES Authors' Group' started by tesAuthorTeam, Jan 19, 2018.

  1. tesAuthorTeam

    tesAuthorTeam Administrator Staff Member TES Authors' forum host

    The purpose of this consultation is to work with the community and use your feedback to introduce ways to drive up resource quality, so that Tes can continue to be a trusted place for teachers to get their resources. For further detail and context please visit the original thread.

    After gathering a lot of feedback, we’ve decided to draw out the main themes that have arisen into separate discussions. This thread focuses on the first theme in our consultation: what quality is and what it means to authors. We’ve had many comments and discussions on this point and would love to hear more ideas and have more debate around this.

    We recognise that defining precisely how to measure the quality of a resource is highly subjective, and have had many debates internally and with teachers about this! In summary, authors have said that at a high-level a quality resource should:
    • Be free from copyright and plagiarism

    • Contain relevant education content and / or up-to-date curriculum standards where applicable*

    • Have been created by and / or tried and tested by real teachers*

    • And if paid-for, should offer good value for money.
    Tes is an enormous library of teaching materials and one we’re incredibly proud to run, but operating and managing such a large platform raises significant challenges.

    Copyright and plagiarism

    There are issues relating to copyright and plagiarism that we are working to address, including looking at how we can provide better support to authors in understanding the rules around copyright and there is much more on this to come. As a community platform we don’t take legal responsibility for the content shared, and operate a takedown policy in the event of any reported infringements. It is important to be realistic that as a community website we cannot implement or fully achieve solutions without your help. There have been comments from authors about having groups of teachers who help with tackling copyright and plagiarism, and we’ll be exploring ideas around this. We appreciate your comments so far and your vigilance in raising concerns.


    Quality stamp or measure

    Several authors including @thinky and @KS2History have suggested a system for ranking resources, which several also agree with but acknowledge that “any such system would need to be very well-thought and transparent.” This is an interesting idea, and we already administer something similar in our search algorithm, where we calculate a specific boost score for each resource appearing in the results. This is based off of a few key factors, including purchases and downloads. However, we’re not sure that this would work in the context as an overall score across the site for each resource, but is only really effective when comparing a group of resources in a specific set of results.

    Others, including @LeafandSTEMLearning, @KS2History, @EC_Resources and @Resource_Creator have suggested a ‘quality badge’ or score connected to sales, reviews or a combination of both.

    This is an idea we’ve discussed and begun exploring here at Tes, both for authors and individual resources; either as an automated system, that could be based on factors like sales (e.g. “Top resource in x this week!”), or as a manual system whereby specific users could review resources or authors and apply a badge. The main issues and points of contention we’ve encountered while talking about these ideas so far has been around how to define what these badges are and when to apply them (what specific measures determine when the badge gets applied), as well as how to ensure that they are properly fair and transparent. We absolutely want to avoid setting up a system that unfairly favours some authors, or that can be easily gamed.

    There are many ways this could be approached, and we’d love to have your take on the matter! So, please tell us what you think.


    Pre-moderation

    Several authors have raised the idea of pre-moderation, including @thinky and @ajs12345 who suggest “vetting new authors to ensure quality is being uploaded”. We agree this could indeed address issues of quality in most cases, though in the spirit of community and as an open platform for resource sharing, we would need to be very careful about this. It’s also a solution that is particularly difficult to manage! It would require a large number of people to put into action effectively, given the number of people that use Tes, and agreeing a clear and uncontentious set of policies on moderating incoming resources or new authors could be somewhat difficult.

    A better solution may lie in more effective processes for reviewing resources and constructive feedback, and there are a number of options here that we’d like to explore. We will cover this in additional threads.

    *Post edited as of 22 Jan to clarify high-level summary of quality indicators, based on summarised community feedback.
     
    Last edited: Jan 22, 2018
    Resource_Creator likes this.
  2. tesAuthorTeam

    tesAuthorTeam Administrator Staff Member TES Authors' forum host

    Authors, on an additional note, if you have been misquoted or we've mentioned you in an incorrect context, please let us know and we'll fix it. Thanks
     
  3. elder_cat

    elder_cat Established commenter

    On the assumption that I am reading it right, I'd be grateful if you could clarify a couple of points for me, using as an example, an imaginary resource on 'how to use HTML to produce a web site':
    • The content may well have educational value, but would be pretty much generic, and might not map directly and explicitly to the standards of any particular curriculum.
    • I am retired, and no longer teach. Therefore, any resources I produce by default will not 'have been tried and tested by real teachers'.
    This appears to be saying that if any resource which is not directly based on 'up-to-date curriculum standards', and has not 'been tried and tested by real teachers', is by default deemed not to be a 'quality resource'.

    This seems to imply that the intention is to restrict future resources to those that:

    a. Are implicitly mapped to specific criteria of current curriculum.
    b. Have been used in the classroom by a practising teacher, and gained their approval.

    Is this a roundabout way of saying that TES feels that a 'closed shop' model, where the only authors allowed to publish resources are those currently in the classroom, whose resources map to what they are currently delivering in the classroom, is the best way to go?
     
    MosaiK, mathsmutt, Kazg1 and 3 others like this.
  4. ResourcesandCourses

    ResourcesandCourses New commenter

    Good point.
     
    MosaiK likes this.
  5. mathsmutt

    mathsmutt Star commenter

    Further, point a. would mean that resources made with the 21CLD aspirations of Apple and Microsoft ( collaboration, knowledge construction, self-regulation, problem-solving and innovation) may not be considered to be quality ones!
    Point b. suggests that the resources submitted have already be made for specific use in the classroom - which means that copyright infringements may exist. Hardly a mark of quality!
     
  6. thinky

    thinky Occasional commenter

    I remember last year there was an initiative to supposedly encourage brand new 'quality' resources where it (was at least suggested) that in return for providing five of 'our best' resources, those resources would be promoted etc. etc.

    When it came to it, it seemed that TES had more submissions than they could manage and only a very small number were given any sort of (limited) promotion.

    Isn't it then the case that TES still has details and examples of what a large number of sellers consider to be some of their best resources, and if so can't these be used? (Maybe even still promoted?!?)

    And doesn't it also suggest that TES must have some definition of quality in order to have judged those resources back then?

    It must also have been a lesson in that managing indicators of quality will inevitably require a large input of time, planning and management?
     
  7. tesAuthorTeam

    tesAuthorTeam Administrator Staff Member TES Authors' forum host

    @elder_cat et al thanks for your comments and queries regarding the original post.

    We have made some edits to the summarised points provided. We hope this helps clarify the above.

    To clarify, our summary of the above points is not intended as a statement from Tes as such but rather an attempt to summarise feedback as succinctly as possible in order to prompt further discussion. We have edited the main post to more broadly summarise the points made and we appreciate your feedback on this too.

    We encourage more authors to submit their feedback and views on these and other topics. We will soon be sharing additional summaries to discuss additional relevant topics.
     
  8. thinky

    thinky Occasional commenter

    I never picked up on these points so I'm not sure of the context but I can't see how being authored by somebody 'in the classroom' can be an indication of quality. I'm impressed at what many teachers do in terms of creating really nice quality resources, however for most teachers I know they don't have the time to put hours into creating a resource (hence why they buy them in the first place!).

    I have the time to research, draft, redraft and make changes. My aim is to provide content that teachers wouldn't have the time to do for themselves.

    In terms of being matched to 'the curriculum'... which curriculum would that be?! This links to the above in that while one teacher in one classroom may have developed the best ever resource for their style of teaching and the particular curriculum they're following... that doesn't mean it will suit somebody else teaching the same curriculum, or of course somebody else teaching a different curriculum in a different part of the world.

    Also worth having in mind that the most enquiries I receive are from home educators and tutors.

    I would argue that more relevant factors would include:
    - adaptability: so they can be easily used by those needing them
    - completeness: providing what is necessary to use them (guidance, solutions, examples)
    - organised: being easy to navigate and use (clearly labelled, in folders)
    - accuracy: of content
    - tidiness: not full of typos, not low quality/poor resolution artwork - prints off/displays on screen nicely

    These can be better described but hopefully makes the point.
     
  9. elder_cat

    elder_cat Established commenter

    Fair enough, thanks for taking the time to respond.:)
     
    tesAuthorTeam likes this.
  10. elder_cat

    elder_cat Established commenter

    Just a thought, but does anyone else think it may be easier to manage if whatever rating system is to be used (stars, etc) is applied to the Authors, rather than to their individual resources? :)
     
    EC_Resources likes this.
  11. tesAuthorTeam

    tesAuthorTeam Administrator Staff Member TES Authors' forum host

  12. elder_cat

    elder_cat Established commenter

    Hi. No, I was just toying with the idea that when someone downloads something (regardless of whether it was a paid for resource, or a free one), they could rate the Author rather than the resource. I just thought maybe it could serve as a way of identifying authors who produce consistently good resources. Over time, that would allow an author to gain a 'reputation for quality', regardless of whether their resources were all free, all paid for, or a mixture of both types. I suppose the old issue arises though, in that it only works if the people downloading can be bothered to tick a box labelled "Would you recommend this author?'
     
    Kazg1 and MosaiK like this.
  13. nwilkin

    nwilkin New commenter

    My idea for a quality mark algorithm: It is quite mathematical, but I think this could work on a computerised system very easily.
    • Each resource is given a starting value of 0.
    • For every view that resource receives from a unique IP address 1 is added to their score
    • For every download this would add 5 to their score.
    • For every positive review (of a 4 or a 5, without comments) they should get the star rating (i.e. a 5 star review will no comment is +5) added to their score
    • If the review has bothered to add comments this would double the score so potentially a 10 could be added for a 5 star review with comments.
    • For any 1 star rating they would get a – 5 and a 2 star rating a – 3 score. With comments these can be doubled.
    • If a users ask for a refund this would give a - 10 score.
    This calculation can then be worked out for all the resources for an author and the mean average taken. Therefor, if an author has a few great resources and some not so good ones it would even out but if they have lots of poor resources (i.e. pages and pages of word searches) then this would bring down their quality average.

    If an author has only one resource which is great this could put them in the top rankings with just that single resource.

    The top 5% of authors with the highest scores could then be assigned a quality badge which they can display on their shop banner and their resource will appear in the “quality” resource search.

    If an author is found guilty of copyright infringement their score should be reset to 0 or the negative value (whichever is lower) the badge should be taken away and they can not get it again for 1 year as punishment for being a numpty. It would also mean they can no longer display the logo in their banner and they will not appear in the “quality” search or on the front page of any resources.

    I know this sounds incredibly complicated but I think it takes into account all sorts of things and could be used as a benefit to authors who are not premium authors but also would be good for those authors who have lots of good stuff out there but are not getting recognised.
     
  14. Krazikas

    Krazikas Occasional commenter

    Reviews are valuable as they give potential purchasers some indication of the quality of the product and hence can increase sales. However, there is so much left to be desired about the current TES review system that I believe any calculations based upon them would be unreliable even if all the data could be pulled together. I don't see the point of doing such calculations.

    To base an author's quality mark on inaccurate, fabricated and the general dearth of genuine and honest reviews would be flawed from the outset . At the moment there are so many errors with current vital systems that I think these should be addressed first.

    Namely:

    1. The sales figures for most bundles are inaccurate.
    2. The algorithm for the search function on TES is not clear and not been explained, despite many requests from authors on the TES facebook author forum.
    3. We do not even have access to our data for conversion rates.
     
  15. elder_cat

    elder_cat Established commenter

    I think there's a lot of things that need to be worked out in terms of how quality is assessed. I understand where you are coming from with your quality mark algorithm, and it's good that people can be bothered to think about possible solutions. But putting my 'devil's advocate' hat on ....

    I don't profess to be a networking guru, but if I understand it correctly , then you have two IP addresses assigned to your laptop or Pc when you conect to the Internet. Your router assigns a private IP address to your laptop, and it also assigns a public IP address as well. The public IP address changes each time you disconnect, or go into standby for some length of time. So which one do we treat as 'unique'?


    This means the reviewer has complete control over how what score the author is awarded, based purely on whether or not they can be bothered (or have the time) to add comments.

    So anyone getting an unjustifiable negative review (ie, troll review) is automatically punished, regardless of whether the actual review has any credibility?

    So someone downloads your resource, and then decides that it isn't what they thought it was (through no fault of the author), and then decides they don't want it. They ask for arefund, and the author is automatically punished?

    Maybe, but the other side to this coin is that if an author has only one resource, and it happens to attract a load of negative (troll)reviews, this would automatically put them at the bottom of the table, through no fault of their own?

    Although there are guidelines published to try and help people with it, making sure you don't fall foul of copyright isn't as clear cut as many people might like it to be.

    Suppose a well-meaning, but misinformed, new author puts a resource up that infringes copyright. Under this system we automatically give them a 1 year 'sentence', without at least giving them the option of correcting their genuine mistake. How many new authors might be suspended from uploading any resources for 12 months? How many might go elsewhere rather than wait 12 months?
     
    MosaiK, hoppytimes and EC_Resources like this.
  16. EC_Resources

    EC_Resources Occasional commenter

    We shouldn't be rating products. We SHOULD be rating authors. Here's why....

    Say I am an unscrupulous person and I upload some lessons which are amazing.... but they're not mine.

    Those products under a product rating system could have amazing ratings - and let's say no one spots that they are copies of another author's work. They continue having amazing ratings and the author gets lots of lovely money because they uploaded another person's amazing work. They could keep doing this and it keep happening.

    If we rate AUTHORS however - upload another person's work and you instantly have a black mark against your name. OK you could open another account - but then you're back to square one in your ratings, or bronze, or whatever we call it.

    These author ratings should be earned - yes that does put established authors at an advantage - but we have all been new authors at some point and it should be clear and easy to work your way up if you play fair and put the hard work in.
     
  17. nwilkin

    nwilkin New commenter


    The IP address attached to your computer, it is the same technology of why Google keep popping up adverts for things you have previously looked at, it is all kept in your digital footprint and TES would know which IP addresses visit their websites and which accounts are sharing IP addresses very easily.

    You can still ask for reviews to be removed and they should not then be included in your score.

    If it is happening a lot i.e. more than once every 10 downloads there is either something wrong with the product or the description. As an author I would want to know that as I would find it concerning if 10% of my sales were being refunded, and I would want to address the underlying problem.


    As I said you can ask for unjustified reviews to be removed, however if a resource is getting a lot of negative reviews there is no way they should be put up high as a quality mark.


    I never said it would stop them from uploading for a year only they just could not get the quality mark for a year. Teachers who are publishing resources for public use have a duty to be aware of copyright and what is and isn’t acceptable. There is no excuse for ignorance in this matter, copyright infringement is illegal and it has the possibility of closing down the entire site because some people are too lazy to find out what is acceptable and what isn’t.
     
  18. mathsmutt

    mathsmutt Star commenter

    This is so true.
     
    MosaiK and Kazg1 like this.
  19. thinky

    thinky Occasional commenter

    An IP address isn't very specific. Cookies are usually used for tracking users but there are ways around all of these things (as I'm sure any secondary student can demonstrate).

    one view on my PC
    one view on my mobile
    one view on my tablet
    one by each of my mates while at the pub... all helping to get me up the ratings etc. etc.

    I think Emily's suggestion is good - some emphasis on rating an author could build credibility. Maybe reviews/feedback could be left with no stars but people could still 'star' the author based on things like originality, usability, value etc. etc.
     
    EC_Resources and mrajlong like this.
  20. elder_cat

    elder_cat Established commenter

    If what you say is correct, then fair enough. I mistakenly
    thought that they tracked you more by the use of cookies and sessions, rather than the IP address, and I'm not sure what would happen in the case of someone accessing TES through TOR?

    The points I raised were related to the very fact that this is meant to work on a computerised system. In other words, the automatic subtraction of points for a perceived infringment would be done automatically based on an algorithm.

    You mentioned the idea of having bad reviews etc 'pulled'. I assume that means contacting TES to have it done. Until that happens, you have been 'tried and found guilty' by the computer system, and the negative effects are damaging your reputation, until such time as TES can undo the damage.

    Absolutely correct, they do. But from what's been aired in a lot of posts, I suspect the real issue with copyright is not one of authors not being bothered to take care of copyright in their own resources. The issue is with others taking their resources and passing them off as their own.

    I agree that some people are lazy. But I can also see others might make a genuine mistake, not because they are lazy, but simply because they have not fully understood all the ins and outs of copyright.
     

Share This Page