RESEARCH PAPERS

Study on Human Preference Evaluation for Text-to-Image

  • Zhao Yimin ,
  • Zhao Yuxi
Expand
  • 1 Library of Yunnan Normal University, Kunming 650500;
    2 Jiangsu Institute of Quality and Standardization, Nanjing 210029

Received date: 2024-07-15

  Revised date: 2025-01-23

  Online published: 2026-01-16

Abstract

[Purpose/Significance] With the rapid development of artificial intelligence, the application of textto-image generation has become increasingly widespread. It is necessary to strengthen the quantitative evaluation research based on human preferences for such images. [Method/Process] The evaluation index system(CDICT) suitable for the characteristics of text-to-image was constructed by the Delphi method, and Midjourney was used to generate image samples embodying different thresholds for each evaluation index, collect preference scores for the measured images globally by an online survey, and analyze the differences in the overall preference and the preference of specific groups. [Result/Conclusion] The human preference of text-to-image is closely related to color performance, detail expression, image processing, creative expression and text-graphic matching, with obvious preference differences in groups and greater influence by hometown culture. Based on this, it propose the image optimization strategies of improving color performance and detail expression, balancing image processing and creative expression, strengthening text-diagram matching and participatory feedback, adjusting the degree of polarization of image features, and paying attention to group differences in image preferences.

Cite this article

Zhao Yimin , Zhao Yuxi . Study on Human Preference Evaluation for Text-to-Image[J]. Library and Information Service, 2026 , 70(1) : 116 -128 . DOI: 10.13266/j.issn.0252-3116.2026.01.010

References

[1] RAMESH A, PAVLOV M, GOH G, et al. Zero-shot text-toimage generation[C]//International Conference on Machine Learning. New York:PMLR, 2021:8821-8831.
[2] DING M, YANG Z, HONG W, et al. Cogview:mastering textto-image generation via transformers[J]. Advances in neural information processing systems, 2021, 34:19822-19835.
[3] ESSER P, ROMBACH R, OMMER B. Taming transformers for high-resolution image synthesis[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition.Piscataway:IEEE, 2021:12873-12883.
[4] GAFNI O, POLYAK A, ASHUAL O, et al. Make-a-scene:scenebased text-to-image generation with human priors[C]//European Conference on Computer Vision. Cham:Springer Nature, 2022:89-106.
[5] DING M, ZHENG W, HONG W, et al. Cogview2:faster and better text-to-image generation via hierarchical transformers[J].Advances in neural information processing systems, 2022, 35:16890-16902.
[6] YU J, XU Y, KOH J Y, et al. Scaling autoregressive models for content-rich text-to-image generation[J]. Transactions on machine learning research, 2022, 2(3):5-54.
[7] NICHOL A, DHARIWAl P, RAMESH A, et al. Glide:towards photorealistic image generation and editing with text-guided diffusion models[J]. arxiv preprint arxiv:2112.10741, 2021.
[8] ROMBACH R, BLATTMANN A, LORENZ D, et al. Highresolution image synthesis with latent diffusion models[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. Piscataway:IEEE, 2022:10684-10695.
[9] RAMESH A, DHARIWAL P, NICHOL A, et al. Hierarchical text-conditional image generation with clip latents[J]. arxiv preprint arxiv:2204.06125, 2022.
[10] SAHARIA C, HO J, CHAN W, et al. Image super-resolution via iterative refinement[J]. IEEE transactions on pattern analysis and machine intelligence, 2022, 45(4):4713-4726.
[11] CHEN J, HUANG Y, LV T, et al. Textdiffuser:diffusion models as text painters[J]. Advances in neural information processing systems, 2024(36):9353-9387.
[12] CHEN J, HUANG Y, LV T, et al. TextDiffuser-2:unleashing the power of language models for text rendering[J]. arxiv preprint arxiv:2311.16465, 2023.
[13] FENG W, HE X, FU T J, et al. Training-free structured diffusion guidance for compositional text-to-image synthesis[J]. arxiv preprint arxiv:2212.05032, 2022.
[14] LIU N, LI S, DU Y, et al. Compositional visual generation with composable diffusion models[C]//European Conference on Computer Vision. Cham:Springer Nature, 2022:423-439.
[15] DENG J, DONG W, SOCHER R, et al. Imagenet:a largescale hierarchical image database[C]//2009 IEEE conference on computer vision and pattern recognition. Piscataway:IEEE,2009:248-255.
[16] LIN T Y, MAIRE M, BELONGIE S, et al. Microsoft coco:common objects in context[C]//Computer Vision-ECCV 2014:13th European conference. Zurich:Springer, 2014:740-755.
[17] REED S, AKATA Z, LEE H, et al. Learning deep representations of fine-grained visual descriptions[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition.Piscataway:IEEE, 2016:49-58.
[18] SALIMANS T, GOODFELLOW I, ZAREMBA W, et al.Improved techniques for training gans[J]. Advances in neural information processing systems, 2016, 29:2234-2242.
[19] HEUSEl M, RAMSAUER H, UNTERTHINER T, et al. Gans trained by a two time-scale update rule converge to a local nash equilibrium[J]. Advances in neural information processing systems, 2017(30):6626-6637.
[20] ZHOU S, GORDON M, KRISHNA R, et al. Hype:a benchmark for human eye perceptual evaluation of generative models[J].Advances in neural information processing systems, 2019(32):3449-3461.
[21] ZHANG R, ISOLA P, EFROS A A, et al. The unreasonable effectiveness of deep features as a perceptual metric[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. Piscataway:IEEE, 2018:586-595.
[22] KIRSTAIN Y, POLYAK A, SINGER U, et al. Pick-a-pic:an open dataset of user preferences for text-to-image generation[J].Advances in neural information processing systems, 2023(36):36652-36663.
[23] XU J, LIU X, WU Y, et al. Imagereward:learning and evaluating human preferences for text-to-image generation[J]. Advances in neural information processing systems, 2024(36):15903-15935.
[24] HESSEL J, HOLTZMAN A, FORBES M, et al. Clipscore:a reference-free evaluation metric for image captioning[J]. arXiv preprint arXiv:2104.08718, 2021.
[25] CHO J, HU Y, GARG R, et al. Davidsonian scene graph:improving reliability in fine-grained evaluation for text-image generation[J]. arXiv preprint arXiv:2310.18235, 2023.
[26] CHO J, ZALA A, BANSAL M. Visual programming for stepby-step text-to-image generation and evaluation[J]. Advances in neural information processing systems, 2024(36):6048-6069.
[27] HU Y, LIU B, KASAI J, et al. Tifa:accurate and interpretable text-to-image faithfulness evaluation with question answering[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Piscataway:IEEE, 2023:20406-20417.
[28] YAROM M, BITTON Y, CHANGPINYO S, et al. What you see is what you read?Improving text-image alignment evaluation[J].Advances in neural information processing systems, 2024(36):1601-1619.
[29] GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al.Generative adversarial nets[J]. Advances in neural information processing systems, 2014(27):2672-2680.
[30] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[J]. Advances in neural information processing systems,2017, 30:5998-6008.
[31] RUIZ N, LI Y, JAMPANI V, et al. Dreambooth:fine tuning textto-image diffusion models for subject-driven generation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2023:22500-22510.
[32] ZHANG H, SHAO H. Exploring the latest applications of OpenAI and ChatGPT:an in-depth survey[J]. CMES-computer modeling in engineering&sciences, 2023, 138(3):2061-2102.
[33] RADFORD A, KIM J W, HALLACY C, et al. Learning transferable visual models from natural language supervision[C]//International conference on machine learning. New York:PMLR,2021:8748-8763.
[34] PALMER S E. Vision science:photons to phenomenology[M].Cambridge:MIT Press, 1999.
[35] LIU J, LUGHOFER E, ZENG X. Toward model building for visual aesthetic perception[J]. Computational intelligence and neuroscience, 2017(1):1292801.
[36] SALEHI S, SHAFIEI M, YEO T, et al. ViPer:visual personalization of generative models via individual preference learning[C]//European conference on computer vision. Cham:Springer, 2025:391-406.
[37] TIAN D P. A review on image feature extraction and representation techniques[J]. International journal of multimedia and ubiquitous engineering, 2013, 8(4):385-396.
[38] NIXON M, AGUADO A. Feature extraction and image processing for computer vision[M]. Academic Press, 2019:12-20.
[39] CHO J, ZALA A, BANSAL M. Dall-eval:probing the reasoning skills and social biases of text-to-image generation models[C]//Proceedings of the IEEE/CVF international conference on computer vision. Piscataway:IEEE, 2023:3043-3054.
[40] SAHARIA C, CHAN W, SAXENA S, et al. Photorealistic textto-image diffusion models with deep language understanding[J].Advances in neural information processing systems, 2022(35):36479-36494.
[41] SHI H, KWOK N, FANG G, et al. Gradient-guided color image contrast and saturation enhancement[J]. International journal of advanced robotic systems, 2017, 14(3):1-11.
[42] ŽEGER I, BILANOVIĆN,ŠIŠUL G, et al. Comparison of metrics for colorized image quality evaluation[C]//2022International Symposium ELMAR. Piscataway:IEEE, 2022:209-214.
[43] RAJEVENCELTHA J, GAIDHANE V H. An efficient approach for no-reference image quality assessment based on statistical texture and structural features[J]. Engineering science and technology, an international journal, 2022(30):101039.
[44] HUNG C C, SONG E, LAN Y, et al. Image texture, texture features, and image texture classification and segmentation[J].Image Texture analysis:foundations, models and algorithms,2019(1):3-14.
[45] DING K, MA K, WANG S, et al. Image quality assessment:unifying structure and texture similarity[J]. IEEE transactions on pattern analysis and machine intelligence, 2020, 44(5):2567-2581.
[46] NIXON M, AGUADO A. Feature extraction and image processing for computer vision[M]. Amsterdam:Academic Press,2019.
[47] LYU Y, WANG X, LIN R, et al. Communication in human-AI co-creation:perceptual analysis of paintings generated by text-toimage system[J]. Applied sciences, 2022, 12(22):11312.
[48] ZHENG M, SIMSAR E, YESILTEPE H, et al. Stylebreeder:exploring and democratizing artistic styles through text-to-image models[J]. arXiv preprint arXiv:2406.14599, 2024.
[49] LEE T, YASUNAGA M, MENG C, et al. Holistic evaluation of text-to-image models[J]. Advances in neural information processing systems, 2024(36):69981-70011.
[50] PARK D H, AZADI S, LIU X, et al. Benchmark for compositional text-to-image synthesis[C]//Thirty-fifth conference on neural information processing systems datasets and benchmarks track (round 1). Cambridge:MIT Press, 2021.
[51] DINH T M, NGUYEN R, HUA B S. Tise:bag of metrics for text-to-image synthesis evaluation[C]//European conference on computer vision. Cham:Springer, 2022:594-609.
[52] JIANG M, HUANG Q, ZHANG L, et al. Tiger:text-to-image grounding for image caption evaluation[J]. arXiv preprint arXiv:1909.02050, 2019.
[53] SANTONEN T, KAIVO-OJA J. The Crowdsourcing Delphi:a method for combing expert judgements and wisdom of crowds[C]//International conference on knowledge management in organizations. Cham:Springer International Publishing, 2022:233-244.
[54] PAVLOVA I, PETRYTSA P, ANDRES A, et al. Measuring physical literacy in Ukraine:development of a set of indicators by Delphi method[J]. Physical activity review, 2021, 1(9):24-32.
[55] MEI W B, HSU C Y, OU S J. Research on the evaluation index system of the construction of communities suitable for aging by the Fuzzy Delphi method[J]. Environments, 2020, 7(10):92-119.
[56] 沈绮云,欧阳河,欧阳育良.产教融合目标达成度评价指标体系构建--基于德尔菲法和层次分析法的研究[J].高教探索, 2021(12):104-109.(SHEN Q Y, OUYANG H,OUYANG Y L. Construction of the evaluation index system for the achievement degree of production and education integration goal[J]. Higher education exploration, 2021(12):104-109.)
[57] LUO W, WANG X, TANG X. Content-based photo quality assessment[C]//2011 international conference on computer vision. Piscataway:IEEE, 2011:2206-2213.
[58] DHAR S, ORDONEZ V, BERG T L. High level describable attributes for predicting aesthetics and interestingness[C]//CVPR2011. Piscataway:IEEE, 2011:1657-1664.
[59] MAVRIDAKI E, MEZARIS V. A comprehensive aesthetic quality assessment method for natural images using basic rules of photography[C]//2015 IEEE international conference on image processing (ICIP). Piscataway:IEEE, 2015:887-891.
[60] ZHOU B, LAPEDRIZA A, KHOSLA A, et al. Places:a10 million image database for scene recognition[J]. IEEE transactions on pattern analysis and machine intelligence, 2017,40(6):1452-1464.
[61] HUTTO C, GILBERT E. Vader:a parsimonious rule-based model for sentiment analysis of social media text[C]//Proceedings of the international AAAI conference on web and social media. Menlo Park:AAAI, 2014, 8(1):216-225.
[62] BROCK A, DONAHUE J, SIMONYAN K. Large scale GAN training for high fidelity natural image synthesis[J]. arxiv preprint arxiv:1809.11096, 2018.
[63] KARRAS T, AITTALA M, LAINE S, et al. Alias-free generative adversarial networks[J]. Advances in neural information processing systems, 2021(34):852-863.
[64] WANG X D, ZHOU X, FATHI A, et al. Visual lexicon:rich image features in language space[J]. arXiv preprint arXiv:2412.06774, 2024.
[65] DEVLIN J, CHANG M W, LEE K, et al. Bert:pre-training of deep bidirectional transformers for language understanding[J].arxiv preprint arxiv:1810.04805, 2018.
[66] BROWN T, MANN B, RYDER N, et al. Language models are few-shot learners[J]. Advances in neural information processing systems, 2020, 33:1877-1901.
[67] CRESWELL A, WHITE T, DUMOULIN V, et al. Generative adversarial networks:an overview[J]. IEEE signal processing magazine, 2018, 35(1):53-65.
[68] FENG F, XIE Y, WANG J, et al. Redefining<Creative> in dictionary:towards a enhanced semantic understanding of creative generation[J]. arXiv preprint arXiv:2410.24160, 2024.
[69] AN J, ZHU Y, LI Z, et al. AGFSync:leveraging AI-Generated feedback for preference optimization in text-to-image generation[J]. arxiv preprint arxiv:2403.13352, 2024.
[70] SHIMODA W, INOUE N, HARAGUCHI D, et al. Type-R:automatically retouching typos for text-to-image generation[J].arXiv preprint arXiv:2411.18159, 2024.
[71] LEPCHA D C, GOYAL B, DOGRA A, et al. A deep journey into image enhancement:a survey of current and emerging trends[J].Information fusion, 2023, 93:36-76.
[72] HONG S. Smoothed energy guidance:guiding diffusion models with reduced energy curvature of attention[J]. arXiv preprint arXiv:2408.00760, 2024.
Outlines

/

〈 〉